Variable Mini-Batch Sizing and Pre-Trained Embeddings
نویسندگان
چکیده
This paper describes our submission to the WMT 2017 Neural MT Training Task. We modified the provided NMT system in order to allow for interrupting and continuing the training of models. This allowed mid-training batch size decrementation and incrementation at variable rates. In addition to the models with variable batch size, we tried different setups with pre-trained word2vec embeddings. Aside from batch size incrementation, all our experiments performed below the baseline.
منابع مشابه
TimeNet: Pre-trained deep recurrent neural network for time series classification
In the spirit of the tremendous success of deep Convolutional Neural Networks as generic feature extractors from images, we propose Timenet : a multilayered recurrent neural network (RNN) trained in an unsupervised manner to extract features from time series. Fixed-dimensional vector representations or embeddings of variable-length sentences have been shown to be useful for a variety of documen...
متن کاملAdapting Pre-trained Word Embeddings For Use In Medical Coding
Word embeddings are a crucial component in modern NLP. Pre-trained embeddings released by different groups have been a major reason for their popularity. However, they are trained on generic corpora, which limits their direct use for domain specific tasks. In this paper, we propose a method to add task specific information to pre-trained word embeddings. Such information can improve their utili...
متن کاملMultiple Batch Sizing through Batch Size Smoothing
Batch sizing in different planning period is categorized as a classical problem in production planning, that so many exact & heuristic methods have been proposed to solve this problem, each of which considering various aspects of the original problem. The solution obtained from majority – e.g. MRP – is in this format that there may be some periods of idleness or each period should produce a...
متن کاملReconstruction of Word Embeddings from Sub-Word Parameters
Pre-trained word embeddings improve the performance of a neural model at the cost of increasing the model size. We propose to benefit from this resource without paying the cost by operating strictly at the sublexical level. Our approach is quite simple: before task-specific training, we first optimize sub-word parameters to reconstruct pre-trained word embeddings using various distance measures...
متن کاملemoji2vec: Learning Emoji Representations from their Description
Many current natural language processing applications for social media rely on representation learning and utilize pre-trained word embeddings. There currently exist several publicly-available, pre-trained sets of word embeddings, but they contain few or no emoji representations even as emoji usage in social media has increased. In this paper we release emoji2vec, pre-trained embeddings for all...
متن کامل